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1. INTRODUCTION 

This paper illustrates error occurrences and drawbacks of using ultrasonic technology for gesture 
control through the project Gesture control for computers using ultrasonic sensors [1]. Earlier interaction 
between computer and human was only limited to keyboard and mouse, but today you can interact with 
gestures i.e facial recognition, and hand gestures [2]. Gestures and image recognition are used in human skin 
detection [3]. Gestures being the primitive form of communication within humans, these are highly useful for 
computer interaction [4], [5]. Dynamic gesture recognition of human gestures makes up an important means 
of user-machine interface [6]. Gesture controls are used in computer vision for human computer interaction 
with electromyography [7]-[9]. Automated recognition of human gesture is a boon for user interfacing [10]. 
Human hand gestures have become more natural and with the help of wireless communication [11], various 
gestures of the computer can be controlled using these sensors and hand recognition [12]. Gestures are 
detected using optical and motion technology by the means of bluetooth, acoustics, tactiles, or infrared 
waves [13]. In the vision based system, motion and skin color information is utilized to detect waving hands 
requesting control commands [14]. Research is still very active in unresolved challenges such as reliable 
input such as identification of gestures, sensitivity to variations in size, shape, speed and issues due to 
occlusion [15]. Making a cost efficient highly accurate model for a gesture control system is essential for the 
future [16]. 
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Vision-based gesture recognition models are based on gesture detection and classification, feature 
extraction and pre-programmed template execution process [17]. Various gestures of the computer can be 
controlled using these sensors and hand recognition. Hand recognition based on visual technology is an 
active area of research from ages [18]-[20] These approaches are based on distance and accuracy which are 
quite flexible, but sometimes they make wrong detection on lighting changes and reflection. There are not 
many practical hand gesture recognition systems that use visual technologies due its inaccuracy despite its 
cost efficiency [21]. One of the prime features of using hand gestures is to interact with the computer as an 
input unit [22], [23]. So, this paper presents the study on the most cost-efficient models and ways to increase 
their accuracy. These models are precisely demonstrated using two techniques namely infrared and ultrasonic 
sensors. 

In the present work, each model was provided with six numbers of gestures to manipulate the screen 
motions and gestures were set and programmed accordingly in the Arduino integrated development 
environment and using python programming language, commands were given to the display [24]. Distance 
between sensor and hand are measured using sensors, which helps to detect the type of movement and then 
the computer shows the result [25]-[27]. Then accuracy of gestures was quantified and represented in pie 
charts using proper data visualization techniques in Jupyter notebook. Differences in accuracy between two 
technologies were quantified and results were concluded. Figure | shows the flow of function in an ideal 
gesture control system. Input is given to the system which is then read by the sensing devices. Then the 
microcontroller interprets and processes the information. After interpretation a command or a signal is given 
to the system. This command or signal interacts with the output or display such that favorable changes are 
made in the display or output. Figure 2 shows the flow of function in a proximity based ideal gesture control 
system. Gesture is given as input to the system which is then read by the proximity sensors. Then the 
microcontroller interprets and processes the information. After interpretation a command or a signal is given 
to the system. This command or signal interacts with the output or display such that favorable changes are 
made in the output. 


SENSING INTERPRETATION CONTROL COMMAND OUTPUT 
DEVICE 


Figure 1. Flow chart for an ideal gesture control device 


INTERPRETED BY PYTHON GIVES SYSTEM INTERACTS WITH 
MOVEMENT CONTROLLER CONTROL COMMAND OUTPUT/DISPLAY ACCORDING 
SENSED BY PSD TO SYSTEM TO COMMAND SYSTEM 
(ARDUINO) 


Figure 2. Flow chart for proximity sensor-based gesture control system 


GESTURE 


2. RESEARCH METHOD 
2.1. Gesture control system using ultrasonic sensor 

Different components are required to develop the gesture control system using ultrasonic sensors. 
Those are Arduino UNOx1, ultrasonic sensorsx2, USB cable (for Arduino), a device for programming and 
display (PC) and jump wires. Figures 3 and 4 depicts the circuit diagram for ultrasonic sensor-based gesture 
control systems. 

The trigger and echo pins of the first ultrasonic sensor are connected with pins 8 and 9 of the 
Arduino while pins 2 and 3 of the Arduino are connected with the trigger and echo pins of the second 
ultrasonic sensor. The two ultrasonic sensors are placed on two corners of the display, i.e., the left and right 
side of the PC display in our case. If they are placed too close then the signal will interfere and wrong results 
will come. After checking the connections, serial inputs are set in the Arduino IDE representing each input 
point in the circuit. In the loop function separate functions for the left, right, up and down gestures are 
defined. The distance is calculated using the principle of reflection of sound. When a sound wave travels 
back to us after reflecting a surface, the distance is given by, 


Distance = Speed x (time/2) (1) 
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Half the total time is taken because sound covers the same distance to travel twice. Now, the speed of sound 
at sea level is 343 m/s. 


Distance = (343/2)xtime  (inm) (2) 
Therefore, 
Distance = (17150) x time  (incm) (3) 


In the next step a particular range is defined at which the hand can move towards and away from the 
sensor. Those distances are taken to be 10 to 30 cm to avoid any other noise or disturbance that could occur 
in the background while using this technology. Now since the boundary at which our system works is 
decided, now the system is set to form our command for the right- and left-hand movements. Commands are 
set in such a way that sensors identify the movement of hands from one sensor to another by using reflected 
sound waves on both the sensors in a given time interval. 


Figure 3. Circuit diagram for ultrasonic sensor-based _ Figure 4. Circuit for ultrasonic sensor-based gesture 
gesture control system control systems 


2.2. Gesture control system using infrared sensor 

Components required to develop the gesture control system using infrared sensors [28] are: Arduino 
UNOx1, 2xIR sensor, USB cable (for Arduino), a device for programming and display (PC) and jump wires. 
Figures 5 and 6 depict the circuit diagram for an IR based gesture control system. passive infrared sensors are 
used for enhancing sensing reliability [29]. 

Associated a jumper wire in the collector pin of the integrated circuit to utilize this sensor as a 
simple sensor. To perceive signal developments utilizing IR sensors, a trigger system is utilized. Let two IR 
sensors be placed on opposite sides of the display and named left-IR and right-IR. Additionally, place your 
IR sensor such that there is around 3 cm of hole between them. Code has been set up on the arduino 
according to the principle of reflection of light. 2 sets of infrared sensors each having a receiver and a 
producing part are prepared. The pins are carefully connected to the arduino circuit board, later these pins are 
set to arduino IDE. Now functions for different gesture motions are declared. The gesture | and gesture 2 are 
programmed in such a way that the sensor recognizes how far can our arms from and away from itself, the 
data from the serial monitor is obtained and likewise it is fed to the python program. Meanwhile the 
programs of gesture 5, gesture 6 and gesture 4 are actuated in such a way that both the sensor checks the 
vicinity of the object to one another, likewise, these sensors now capture the wave movement of the hand. 


2.3. Programming Arduino to detect gestures 
Different cases are made to distinguish gestures from each other based on the distance measured by 
the sensors over a time period which is unique to a particular gesture. Creating different cases for different 
gestures makes it possible to execute different commands on sensing different gestures which is done by the 
help of python. Following are the cases: 
— Gesture 1: Place your palm in front of the right sensor and move towards yourself (distance of palm 
from sensors should be between 15 to 35 cm). This motion will scroll down your page in the browser or 
decrease the volume in the media player. 
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— Gesture 2: Place your palm in front of the right sensor and move towards sensors (distance of palm from 
sensors should be between 15 to 35 cm). This motion will scroll up your page in the browser or increase 
the volume in the media player. 

— Gesture 3: Swipe your hand in front of the right ultrasonic sensor. This motion will change your current 
tab to the Next Tab or fast forward the video running in the media player by 10 seconds. 

— Gesture 4: Swipe your hand in front of the left ultrasonic sensor. This motion will change your current 
Tab to the previous tab or backward the video running in the media player by 10 seconds. 

— Gesture 5: Swipe your both palms in front of both sensors simultaneously (left palm a little bit fast). 


Figure 6. Circuit for IR sensor based gesture control system 


This motion switches between tasks (i.e., browser and media player). The automation code for the 
software actuation is programmed in python, different codes are used for different software which are 
internet browser and video player respectively. In python code PyAutoGUI is used to read the commands of 
the arduino IDE provided to the Python code [30]. 

— if ‘next’ is given as input the system produces pagedown activity. 

— if ‘previous’ is given as input the system produces pageup activity. 

— if ‘down’ is given as input the system produces scroll down activity. 

— if ‘up’ is given as input the system produces scroll up activity. 

— if ‘change’ is given as input the system produces change tab activity in the browser. 

Control gestures used for the video player are the same as browsers but have different executions. 
The same PyAutoGUI is used for controlling the output from arduino IDE as well as managing the hotkey 
functions in the keyboard. The gestures are given below: 

— if ‘next’ is given as input the system produces a move forward signal for 10 seconds activity. 

— if ‘previous’ is given as input the system produces a move backwards signal for 10 seconds activity. 
— if ‘down’ is given as input the system produces a decrease in volume activity. 

— if ‘up’ is given as input the system produces an increase in volume activity. 

— if ‘change’ is given as input the system produces a pause activity in the video player. 

Figure 6 depicts the arduino IDE serial monitor. In Figure 7, this window acts as an interface 
between the device and the software to exchange information and the message flow to one another through a 
USB pin. The meaning of the statements in the interface are: 

—  b‘I\r\n’ indicates true value or positive value 
—  b‘0\r\n’ indicates false value or negative value 
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Suppose, for calculating the probability of error for Gesture 1; the output related to Gesture | is considered as 
‘1’ and other undesired outputs that are related to other gestures are considered as ‘0’. When the program is 
executed it automatically detects ‘1’ and ‘0’, from there a dataset of these values is formed. Some analytics is 
performed to find probability and to make the pie charts. This process is done again for all the gestures. 


& *|DLE Shell 3.9.1* 
File Edit Shell Debug Options Window Help 
Python 3.9.1 (tags/v3.9.1:le5d33e, Dec 7 2020, 17:08:21) 


"copyright", "credits" or "license()" for moy 


= RESTART: C:\Users\dsuk1\Downloads\chrome 


>'O\r\n' 


Figure 7. Arduino IDE serial monitor 


3. RESULTS AND DISCUSSION 

Pie charts showcasing accuracy and faultiness of the models against various gestures were made 
using data visualization on the observations that were inferred from our models. The gesture control models 
were experimented on various gestures explained above so that data sets of accurate and faulty commands 
were obtained using an auto checking program made using serial monitor of arduino IDE. Using this data set, 
data visualization was done on it using the jupyter notebook. 


3.1. Ultrasonic sensor 

Experiments performed on ultrasonic sensors showcase the following results. From the 
observations, it is concluded that using ultrasonic sensors for right swipe gesture 66.3% accuracy and 33.7% 
faulty observation as shown in Figure 8; for left swipe gesture 85.9% accuracy and 14.1% faulty observation 
as shown in Figure 9; for scroll down gesture 80.4% accuracy and 19.6% faulty observation as shown in 
Figure 10; for scroll up gesture 72.9% accuracy and 27.1% faulty observation as shown in Figure 11. The 
right swipe gesture as shown in Figure 8 was optimized and it’s accuracy was increased by using a barrier 
between both the sensors as a result, an accuracy of 83.9% was obtained with a 16.1% faulty observation, 
which is described in Figure 12. 


3.2. Infrared sensors 

Experiments performed on infrared sensors showcase the following results. From the observations, 
it is concluded that using infrared sensors for left swipe gesture, 96% accuracy and 4% faulty observation as 
shown in Figure 13; for right swipe gesture, 93.5% accuracy and 6.5% faulty observation as shown in 
Figure 14; for scroll up gesture, 95.5% accuracy and 4.5% faulty observation as shown in Figure 15; for 
scroll down gesture, 94.5% accuracy and 5.5% faulty observation as shown in Figure 16 were obtained. The 
accuracy percentage of two gesture control models namely, ultrasonic sensor based and infrared sensor-based 
system for various gestures were observed. Results also show that putting a barrier in between the sensors 
increases the accuracy of ultrasonic sensors. 

As per above results, the following observations were inferred. Ultrasonic sensors were used 
initially and their observations were analyzed, from which an average accuracy was calculated to be 76.37%. 
Next, the observations were made upon infrared sensors and average accuracy was calculated as 94.87%. 
Also, barrier addition resulted in increasing the accuracy of right swipe observations in ultrasonic sensors to 
about 17.6% of the original value. 
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4. CONCLUSION 

It has been quantified that IR sensors have a better accuracy in Gesture control technology than 
ultrasonic sensors. The increase in accuracy has been found out to about 27.874% through our experiment. 
Among the two technologies, complimenting the gesture motions, studied and compared thoroughly, error 
observations were noted, which were consolidated into the following conclusions: Cons in using ultrasonic 
sensors: Its range is up to 400 cm. If multiple sensors are put closely, there will be interference and it will 
give unwanted output which is unreliable. Moreover, if there is sound absorbing or reflecting material worn 
by the user or are in very close proximity of the sensors the readings will be faulty resulting in wrong 
commands given to the system. Cons in using IR sensors: It gives inaccurate reading if encountered with a 
reflective surface. If the user wears something that is reflective in nature or absorbs the beam of IR, the 
reading of the IR sensor can become inaccurate. If multiple sensors are put closely, there will be interference 
and it will give unreliable output. It can maintain high accuracy for a range of 150 cm after that irregularities 
increase in the readings from the sensor. Ultrasonic sensors work using the principle of sound waves. Their 
results can be affected due to many factors such as noise, other obstacle interactions, sensor liabilities. But 
can be used where there is less sound medium based noise and more heat related noise. Infrared, on the other 
hand, is a more accurate technology considering the gesture technologies and its application. It still has 
shortcomings; for instance, if there are many heat sources in the environment. Error arising in an ultrasonic 
sensor due to noise and other matters can be decreased by adding a barrier in between the ultrasonic sensor to 
decrease the interruption by the sound noises normal to the barrier and isolate the direction to be sensed. 
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